Big Scientific Data Benchmarks, Architecture, and Systems by Unknown

Author:Unknown
Language: eng
Format: epub
ISBN: 9789811359101
Publisher: Springer Singapore

1 Introduction

There are a lot of tricks in machine learning application to get higher training efficiency, better classification accuracy and the ability of solving unconvex optimization. Some of them are reasonable and well-proved, like setting a better initial model parameters to reduce training time. But most of other tricks are lack of proof. Most of those tricks can only be used suitably depending on users’ experience, like deciding the size of batch and constructing a DNN. In a real situation, the majority of tricks are proved by experiments instead of rigorous mathematical proofs.

Nowadays, Parameters Server frame, based on delayed SGD algorithms, is the most popular learning frame. However, with the increasing number of workers, the burden of network would be unaffordable. Asynchronous FTRL-proximal and addressing L2 norm on server are two widely used tricks to solve this problem, but they are not rigorously proved.

Above two tricks share the same commonness. They divide updating computation into two parts. One part is computed at worker. The work of this part is scanning dataset, computing the gradient of loss function without regularization term and sending this portion of loss function gradient, a sparse vector, to server. Another part is computed at server. The work of this part is computing the gradient of regularization term and updating model parameters lazily. These two parts are computed asynchronously and separately on servers and workers. Sparse data vectors in first part reduce the burden of network. Hereafter, these two tricks will be abbreviated as asynch-FTRL-proximal and L2 norm trick.

Based on above commonness, we propose and prove asynchronous Composite Objective MIrror Descent, abbr. asynch-COMID in this paper. Then, we establish the equivalence between asynchronous COMID and the above two tricks to prove above two tricks are applications of asynch-COMID. Thus, the convergences of above two tricks are also proved. We fill these gaps between application and theory of above two tricks via asynch-COMID.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.